In recent years, companies are continuing to build more complex software systems that may include client applications, server applications, and developer tools, all of which need to be supported on multiple hardware and software configurations. This is compounded by the need to deliver high quality applications in the shortest possible time, with the least resources and often involving geographically distributed organizations. Having sensed these realities and complexities, companies are increasingly resorting to writing the applications in Java/J2EE.
Although Java is based on the “write once, run anywhere” paradigm, quality assurance (QA) efforts are nowhere close to the “write tests once and run anywhere” because modern-day software applications still must be tested on a great number of heterogeneous hardware and software platform configurations. Some companies have developed internal QA tools to automate local testing of the applications on each platform, but completing QA jobs on a wide array of platforms continues to be a large problem.
Typically, multi-platform software testing requires a great amount of resources in terms of computers, QA engineers, and man-hours. Because the QA tasks or tests are run on various different types of computer platforms, there is no such point of control, meaning that a QA engineer must first create an inventory of the computer configurations at his or her disposal and match the attributes of each computer with the attributes required for each of the test jobs. For example, there may be various computers with different processors and memory configurations, where some operate under the Windows NT™ operating system while others operate under Linux and some others operating under other UNIX variants (Solaris, HPUX, AIX). The QA engineer must manually matchup each test job written for specific processors/memory/operating system configurations with the correct computer platform.
After matching the test jobs with the appropriate computer platform, a QA engineer must create a schedule of job executions. The QA engineer uses the computer inventory to create a test matrix to track how many computers with a particular configuration are available and which tests should be run on each computer. Almost always, the number of computers is less than the total number of test jobs that need to be executed. This creates a sequential dependency and execution of the tests. For example, if one test completes execution in the middle of the night, the QA engineer cannot schedule another test on the computer immediately thereafter because the startup of the next test requires human intervention. Therefore, the next test on this computer cannot be scheduled until the next morning. In addition, this guesswork for the completion time for the test jobs does not always work because the speed at which the test executes depends on many other external factors, such as the network. One can visualize the difficulties of scheduling and managing the QA tests if there are thousands of tests to be run on various platforms.
Once the jobs are scheduled, the test engineer must then physically go to each computer and manually set up and start each test. Once the tests are in progress, one must visit each of computers in order to check the current status of each test. This involves a lot of manual effort and time. If a particular test has failed, then one must track down the source of the failure, which may be the computer, the network, or the test itself. Because QA engineers are usually busy with other meaningful work, such as test development or code coverage, when the tests are being executed, the QA engineers may not attend to all of the computers to check the status of the tests as often as they should. This delay is the detection and correction of the problems and increases the length of the QA cycle.
This type of manual testing approach also curtails the usage of computer power. Consider for example a situation where a test engineer must run five tests on a particular platform and only has one computer with that configuration. Suppose that the first test last for eight hours. The QA engineer will usually start the first job in evening, so that he has the computer free to run the other tests during the day. If the first test hangs for whatever reason during the night, there's no way to QA engineer will realize it until the morning when he goes back to check the status. Therefore many wasted hours pass before the tests can be restarted.
Because a test may fail several times, the execution of the test finishes in several small steps making the reconciliation of tests logs and results a tedious and time-consuming process. At the end of the test cycle, one must manually collect the tests logs and test results from each of the computers, manually analyze them, and create status web pages and file the bugs. This is again a very tedious and manual process.
What is needed is a test system that manages and automates the testing of software applications, both monolithic as well as distributed. Basically, the test management system should enable the “write once, test everywhere” paradigm. The present invention addresses such a need.